Recently plenoptic or light-field imaging has entered into the consumer domain with the introduction of cameras from manufacturers such as Lytro and Raytrix. Plenoptic imaging offers the potential to refocus a digital image after acquisition.
While these techniques capture additional scene depth and lightfield information over a conventional camera, they generate complicated datasets which are significantly larger than conventional images and require sophisticated post processing to regenerate a conventional image with different parameters. Furthermore they are incompatible with existing image compression techniques and require proprietary viewing software. Thus the benefits of such advanced imaging techniques cannot readily be enjoyed on conventional consumer devices such as smartphones and tablets, or viewed using Web/Internet technologies.
FIG. 1 shows a typical scene including two faces 10, 12; and FIG. 2 illustrates the operation of a prior art auto-focus algorithm for the scene as a function of distance from the camera. This is a conventional “hill-climbing” algorithm which captures an initial preview image of the scene and calculates a focus measure on one or more portions of the acquired image. Note that modern imaging devices automatically perform functions such as face-tracking and foreground/background separation on the individual frames of the preview stream and thus the focus measure is generally not calculated over the entire image, but rather on specific regions of interest (ROI) such as faces and foreground objects within the image.
In FIG. 2 we show the focus measure for the scene as calculated over the area involving the near face 10 of FIG. 1 (steps 1-4) and also the focus measure as calculated over the mid-distance face 12 of FIG. 1.
The simplest focus measure will just measure the local contrast and/or sharpness over the selected ROI of the image. However there are more sophisticated focus measures available in the literature.
It will be appreciated that there are many available auto-focus algorithms described in the literature, and hill-climbing algorithm of FIG. 2 is just a simple example. Note that the focal distance, shown along the X-axis of FIGS. 2, 3 & 5, is on a logarithmic scale. This is typical of the focus adjustment where there are more focus steps close to the camera, and beyond a distance of 3-5 m the camera is effectively focused at infinity.
In FIG. 2, the camera first focuses at distance “1” which is about 18 cm from the camera. This provides a very weak focus measure over the ROI of the near face, so the distance is adjusted to about 35 cm which yields a stronger focus measure, ‘2’, but still below the desired threshold (c. 0.9); a third adjustment, ‘3’, to 65 cm yields a higher focus measure, but based on the size of the face region the algorithm realizes that it has over-shot and re-adjusts the distance to just below 50 cm which is close enough to yield a focus measure at, or above the required threshold.
Now this example is a very simple one, and typically the auto-focus algorithm may take several additional focusing steps to arrive correctly at an optimal focus point. The algorithm is also dependent on the ROI selected to calculate the focus measure, the accuracy of this ROI and any additional information available. In the simple example provided above it was assumed that the focus algorithm would know that step “3” was beyond the optimal focus distance based on a knowledge of the size of the face ROI. If this information was not available, the algorithm would have continued to adjust the focus to a greater distance, and only on determining that the focus measure had decreased (step 4) would it have realized that the focus adjustment should have been to a distance between “3” and “2” (step 5).
Typically, therefore, an auto-focus process will take more steps than shown here. Nonetheless, at the end of the process a single main, in-focus, image is acquired.
In a video system, auto-focus operates a bit differently. As every frame is saved by a video acquisition process the focusing algorithm has two main stages.
In the first stage the focusing algorithm behaves essentially as for the digital camera auto-focus algorithm and the focus may change by quite large shifts in distance until a focus measure above the required threshold is achieved.
After this initial focus is achieved the video camera will continue to adjust the focus, but in smaller increments. In the context of the example above, after the initial focus one each following frame a focus change of only one step +/− is allowed. Thus if the focus measure on the near face 10 drops below the threshold it is only possible to adjust by one step nearer, or more distant from this initial focus on the next image frame. (Or if the frame rate is high, or the focus system has high inertia it may only be possible to change focus every 2-4 frames); the video algorithm must limit focus changes in this way to avoid “focus jumping” or “focus hunting” effects which can easily occur due to the complex and constantly changing nature of video scenes.
Thus video focusing comprises an initial auto-focus phase, which allows large focus adjustments until an initial focus is achieved. This is followed by a second “small adjustments” phase to allow focus to track the main ROI(s) within a scene without making sudden jumps in focus that would be disconcerting to the viewer.
Separately, focus stacking is a technique used in photography to capture multiple images at different focus lengths and combine these into a single “in-focus” image. The technique is typically applied to macro photography for objects close to the camera lens and focus increments may be only a few centimeters and over a range of focal lengths of less than 30-50 cm.
US 2010/0283868, Clark & Brown discloses a system for taking motion pictures of a scene with different parts at different focal lengths with a camera that normally would not have sufficient depth of field to capture all parts of said scene in focus. A computer controls a high-speed, digital camera and a lens with rapid focus capability to produce a stack of images at different focal lengths for rapid processing into a composite serial image montage stream in motion picture output format. Operator controls permit continuous selection of depth of field (DOF) in the output image and enable a plurality of in-focus and out-of-focus regions over the depth of an image. An optional output port provides for real-time recordation of all images in each stack for later processing. An optional remote control duplicates the main controls in the camera system so that a second person can assist in optimizing images as they are recorded, or for remote control of the camera itself.
It is an object of the present invention provide an improved method and apparatus for viewing stacked images.